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Introduction 

When staff and board members of the Dayton Mediation Dayton Mediation Center — a 
publicly-funded agency that offers the services of trained volunteer mediators to people in the 
Dayton, Ohio community — \vanted to define what a “successful” mediation would look like, they 
turned to a process called “action evaluation.” By going through this process, the Mediation 
Dayton Mediation Center hoped to quantify the benefits they knew they’d been providing to the 
community — and thus be able to further promote and expand the reach of their services. 

After meeting with an action evaluator, however, the “stakeholders” decided to focus first 
on clarifying their own organizational goals and priorities. Thus, before addressing the question of 
a successful mediation, they needed first to take a look at the organization itself, to think about 
where it had been, and where it needed to go. And to do this, they turned again to action 
evaluation. 



Measuring the success of any process aimed at changing the way people relate to one 
another can be tricky business. Yet in fields such as conflict resolution and organizational 
development, those directly involved — as well as funders and the general public — are more and 
more often asking for rich definitions of success and for valid and rigorous assessment of it, 
primarily as a means towards accountability. At the same time, practitioners are trying to figure out 
how to inform the public, the funding community, and policymakers about the nature and long- 
term impact of their work. Researchers too are asking questions about assessment. How can it be 
more dynamic and integrated into practice? How can rigorous research contribute to the 
effectiveness of processes? Can it do so without violating issues of confidentiality or imposing 
control on real-life situations that rarely lend themselves to experimental conditions? 

Since 1992, dozens of students, researchers, and practitioners worldwide have collaborated 
on a project to address these complex and interrelated questions. Initially funded by the Pew 
Charitable Trusts, the Action Evaluation Research Initiative resulted in a methodology called 
“action evaluation,” which employs evaluation as a bridge between research and practice in ways 
that promote greater quality in both. 

At first focusing primarily on the field of conflict resolution, action evaluation has become 
an integrated research and intervention methodology for helping to define, promote, and assess 
success in conflict resolution processes, organizational and community development efforts, and 
the creation and enhancement of educational programs. The Action Evaluation Research Initiative 

focused on the development and application of a rigorous evaluation methodology, intended to help 
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project organizers, facilitators, participants, and funders interactively define their shared goals as a 
project evolves, and effectively monitor and assess them. Now called the Action Evaluation 
Project, the ongoing work of refining the action evaluation methodology has been funded since 
1998 by the Hewlett Foundation. 



The Action Evaluation Process 

Relying in large part on a computer assisted and Web-based instrument and database 

system for ongoing data analysis and program monitoring, action evaluation includes: 

• A baseline stage that includes a systematic process for cooperative goal setting, team-building, 
and participatory decision making within and between various project stakeholding groups. 

• A formative stage, during which participants refine their goals, make them more “actionable,” 
and begin working toward them. 

• A summative stage, when participants take stock of their progress, using their evolved goals to 
establish criteria for retrospective assessment. 



Baseline Stage 

The action evaluation process is facihtated by a specially trained “action evaluator” who 
collects data on goals from the various stakeholders and summarizes this data in terms of what is 
shared, unique, and contrasting within and, later, across stakeholding groups. In the case of 
the Dayton Mediation Dayton Mediation Center, there were two “stakeholding groups” — one 
including both staff and board members, the other consisting entirely of volunteer mediators. 



Data gathering 

In the first phase of an action evaluation, all those involved with the project at hand 
articulate what is referred to as the “baseline,” or initial and perhaps rather general, goals. This data 
is gathered via on-line questionnaires (respondents go to the Action Evaluation Project Web site 
and locate a questionnaire designed for their project); through interviews (which can also be used 
to supplement data gathered on-line); or through paper and pencil surveys. The preferred method is 
the on-line computer process since it eliminates a data-entry step and makes the questionnaire both 
interactive and easily accessible from anywhere in the world where the Internet is available. 
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In addition to narrowing geographic gaps and aiding in data gathering and analysis, the 
computer-assisted goal-setting process is designed to be both user friendly and self referential. The 
Web technology allows respondents to refer back to their previous responses as they continue to 
move through the questionnaire. 

The data-gathering process begins with three basic questions: WHAT? WHY? and HOW? 

WHAT goals do the stakeholders have for this initiative? Another way to think about this 
question is to pose visions of success against current reality. The people involved may be asked to 
consider what they hope will change for participants — and in the larger social setting — due to the 
intervention and their involvement with it. 

WHY do the various stakeholders care about their goals? What motivations are driving 
their outcome goals? More conceptually, and directed toward the conveners more than participants 
or funders, what are the theories of practice and domain assumptions that guide their practice? 

HOW might the stated goals be met most effectively? What intervention processes should 
be used? Based on the goals and motivations articulated, stakeholders are asked to suggest what 
kinds of intervention strategies might best encourage movement from the present reality to the 
vision of success they have just articulated. 

Data analysis 

Once everyone has responded to the baseline questions, the action evaluator organizes and 
analyzes the data of each stakeholding group on-line (using a password to protect confidentiality), 
synthesizing the questionnaire responses in a “pre-feedback” page as follows: 

• The WHAT responses from everyone in a given stakeholding group is organized according to 
shared goals (of two or more respondents); unique goals (that only one respondent has 
articulated); and contrasting goals (between two or more respondents). For shared goals, 
the action evaluator may summarize, combine, or otherwise edit responses, as well as list each 
person’s version of the overlapping goals while preserving the original voice of the 
respondent. 

• The WHY responses are simply listed verbatim. 

• The HOW data is listed and quantified according to each specific suggestion and the numbers 
of respondents who have proposed similar ideas, to be later matched with the shared WHAT 
goals. 
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Feedback 



Once the data is compiled, it is printed out and shared with respondents within each of the 
stakeholding groups during face-to-face meetings facilitated by the action evaluator. While the 
compiled data reflects the voice of the respondents it preserves their anonymity. In addition to 
ensuring that the action evaluator gets his or her analysis “right,” these feedback sessions help 
ensure that key project leaders, participants, and others are “on the same page” about their goals as 
they move ahead. 

Feedback sessions begin with participants presenting their “why” data. Not surprisingly, 
they often want to begin with the “what,” but the process begins with “why” for a reason. It forces 
the stakeholders to go deeper, to look at their motivations, their values, as they begin to explore 
their own — and their group’s — goals. The process also enables project members to learn more 
about each other and about their shared and differing motivations in a safe and confidential 
environment. 

Next, the action evaluator presents the compiled “what” data, inviting the group to check 
the re-articulation of the shared goals and clarify whether those listed as unique and contrasting 
goals are correctly interpreted. 

While action evaluators regularly seek group consensus on goals, it is almost always 
lacking at the start of the feedback process. So following the presentations of “why” and “what” 
data, the group negotiates until they arrive at a consensus on shared goals and priorities. With a 
new understanding of the motivators, values and interest involved, the action evaluation process 
helps various interest groups define their shared project goals, as well as employ creatively the 
diversity of their differences and divisions to enrich their efforts. 

Ultimately each stakeholder group comes to a notion of a shared agenda, grounded in their 
common goals and values, for WHAT should be accomplished and WHY it is valuable for it to be 
accomplished. 

After each stakeholding group completes this process, the action evaluator compares the 
shared goals of the different groups. Then the process is repeated with all stakeholding groups 
together, and they translate the HOW data into action steps related to the agreed-upon goals. 
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Refining the Davton Mediation Dayton Mediation Center’s Goals 



When the Dayton Mediation Center staff/board group, which included nine participants, 
convened for their feedback session, the evaluator began by asking each group member to present 
his or her own “why” responses. To do so, they were given a choice: either read your “why” 
responses directly from the compiled data, paraphrase it, or present and embellish 
extemporaneously. In this way, the group very quickly began moving to a deeper level. 

While the written responses tended to be carefully considered explanations for what the 
participants felt needed to be done, the face-to-face presentation enabled them to go deeper, to 
share their gut level feelings about what was most important. For example, one concern of Tom, 
the Dayton Mediation Center Coordinator arose from the fact that the Dayton Mediation Center, 
which is funded by the city of Dayton, serves only the population of the city itself, but not of the 
surrounding metropolitan area. Tom wrote, in part. 

The Dayton Mediation Center needs to grow to survive. The city is losing population. 

Many county officials and citizens value ADR services so the time is right to pick up this 
market. . . . Since mediation is rapidly increasing in use, the time is ripe to develop a plan 
to increase our market share as well as multiply the sources of our referrals. . . . The 
Dayton Mediation Center, if it is to survive, must show that the community values its 
services enough to support it. 

Expressing the same concern, in the meeting he told the group. 

We have to make our own way and we have to serve this community in a way that makes 
us a vital service. If we rest on our laurels from yesterday, this place will be gone 
tomorrow. And I think it ought to be gone tomorrow if it can’t continue to serve this 
community in a vital way. . . . There is a whole community out there that we are not 
serving so we’ve got to get out there and serve them. If we believe that mediation is vital to 
the greater community, not just Dayton, then why not offer this wonderful service? . . . 

The Dayton Mediation Dayton Mediation Center does an incredible job serving this 
community — it almost takes my breath away — I think we have the ability to really affect 
this community, and we are not doing it, just not doing it. . . . 

While the basic meaning of Tom’s comments doesn’t change, the “feeling” of the 

comments does. His writes of “market share” and of “multiplying the sources of our referrals.” 

But he speaks of “a whole community out there that we are not serving” with “this wonderful 

service.” It’s clear from the director’s spoken response to the “why” question that he cares deeply 

about the community, and about the impact — and potential impact — the Dayton Mediation Center 

can have on its people. 

The time they spend articulating their “why” responses also gives participants a chance to 
respond to and build on one another’s concerns, in a manner that’s impossible with the one-way 
responses entered into the project database. Following Tom’s presentation, George, a businessman 
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who serves on the Dayton Mediation Center’s advisory board, gave his responses. His concerns 
revolved primarily around issues of fiscal responsibility and accountability: 

the Dayton Mediation Center needs to be strategic and financially stable to survive 

Mediation service provides a valuable service to the community and in order to continue to 
provide that service we have to be fiscally sound. Mediation is an efficient way of 
achieving a harmonious society. 

The next two speakers acknowledged the importance of financial stability, but they also 
brought their own experiences and perspectives to the issue. Cherise, who recently joined the 
Dayton Mediation Center staff, having recently completed a masters degree in conflict resolution 
from nearby Antioch University. She offered this observation: 

The Dayton Mediation Center is in a unique position to reduce the unnecessary conflicts 

created by segregation and bridge cultural differences Helping citizens understand 

cultural diversity can help them view situations and others differently. The Dayton 
Mediation Center needs to be financially stable but it also needs to get into the community 
and identify the needs of the residents and raise awareness. 

Janet, a contract employee who has worked for the Dayton Mediation Dayton for a number 
of years, added her concerns that have grown out of several years of contact with volunteers and 
with those who have used the Center’s services: 

Volunteers are a tremendous resource and key to being a community mediation.... [its] 
uniqueness permits some individuals to use mediation before violence occurs yet many 
clients do not reach the Dayton Mediation Center until after an occurrence of violence. 
Financial stability would enable the Center to increase and retain staff, thus enabling the it 
to increase the volume of services. 

George, Cherise, and Janet all shared the goal of financial security for the Mediation 
Dayton Mediation Center, but each came at the goal from a different place. What’s important is that 
instead of focusing on their differences, they were able to focus on what they shared: a desire for 
the Dayton Mediation Center to thrive. 

Once all participants had verbalized their “why” responses and had had a chance to respond 
to each others’ concerns, the group turned to the “what” data. On flip charts, the action evaluator 
presented the compiled “what” data, which consisted of four shared goals, each initially 
proposed by four or more group members. As synthesized by the action evaluator, these goals 
were: 

1 . To establish financial stability/seek increased funding. 

2. To expand number and variety of services, and extend them both regionally and to other 
types of markets. 

3. To enhance staff training and opportunities. 

4. To increase referrals, visibility, and usage of Dayton Mediation Center. 
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The evaluator also reported the following unique goals; 

1. To increase and maintain an active, knowledgeable board. 

2. To establish and follow a focused mission policy. 

3. To research and publish Dayton’s conflict resolution needs. 

Finally, the evaluator interpreted the following as contrasting goals: 

1. To seek financial stability for existing services and programs VS. seeking to expand and 
extend services (which would initially cost more than current budget allows). Grant 
support VS. city, corporate or citizen support (Market research and marketing VS. grant 
writing) 

2. Maintain current services VS. expand and extend services and programs offered (for other 
than financial reasons). 

3. Full-time staff VS. part-time or volunteer. 

Following the action evaluator’s presentation, the group worked through the “what” goals 
together, refining and clarifying the evaluator’s interpretations. During this give-and-take session, 
it became clear that some goals that the evaluator had interpreted as contrasting, were not really so. 
In the case of the first “contrasting goal,” for example, the group felt that financial stability for 
existing services and programs was not at odds with the notion of expanded and extended services. 
Rather, they seemed to feel the two goals would support each other: greater financial stability 
would enable them to expand services, while expanded services would enable them to increase 
revenues. The group also rejected the notion that seeking grant support would preclude them from 
utilizing city, corporate, and citizen support. Rather, they saw a need for balancing their sources of 
support: the key was not to rely to heavily on any single source. 

Once all goals were articulated to the group’s satisfaction, they “voted” on them. For each 
goal, participants were asked to give a “thumb up” to indicate full support, as well as a willingness 
to work toward meeting that goal; a “thumb down” to indicate non-support; or a “thumb sideways” 
to indicate some ambivalence, perhaps a recognition that a goal may be significant, but not 
important enough to devote scarce resources to at the current time. 

Ultimately, the board/staff group reached consensus (no thumbs down) on the following 
six goals: 

1 . To establish financial stability/seek increased funding. 

2. To extend services regionally and to other types of markets. 

3. To enhance staff and volunteer training and opportunities. 

4. To increase referrals, visibility and usage of Dayton Mediation Center. 
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5. To increase and maintain an active, knowledgeable board. 

6. To make research part of our practice. 

The evolution of some of the goals illustrates the importance of bringing a group to the 
table, rather than having one or two people try to set goals for an entire organization. For example, 
“to enhance staff training and opportunities” became “to enhance staff and volunteer training and 
opportunities.” The change came about in large part because one of the staff members — actually a 
contract employee — spent much of her time working directly with the volunteer mediators. From 
the perspective she brought to the group, the volunteer mediators stood to gain as much from 
additional training and other opportunities as did the paid staff. Other goals, initially “unique” to 
only one stakeholder, are added to the list of shared goals as the group discusses them. Such was 
the case with “to increase and maintain an active, knowledgeable board.” While this goal was 
brought to the group by one member of the board, everyone — board and staff — could see it as 
essential to the long-term success of the organization. 

A Slight Change of Plans 

One aspect of action evaluation that makes it so appealing, and so useful, is its flexibility. 
Although there is an overall structure to the process, there is room for adaptation to the context and 
progress of a given project. The group took advantage of this flexibility in two significant, inter- 
related ways during their first feedback session. 

First, the original agenda for the session included a one-hour session on “next steps.” At 
this time, the group was to have matched “how” suggestions to “what” .goals. Then the action 
evaluator would have introduced the group to “action planning,” the process that would begin at 
the group’s next meeting. However, during the course of the 4-hour session, it became apparent 
that the group needed to spend more time on their “why” presentations and then on negotiating 
consensus for their “what” goals. 

Even more significantly, the group — which included both staff and members of the board 
— begtui to consider whether a second stakeholding group, made up of volunteer mediators, 
should be consulted. The group worried about the appropriateness of involving volunteer in policy 
making activities, whether the volunteers would be interested in participating, if the volunteers had 
anything to contribute, the difficulty of assembling volunteers on short notice, and they worried 
about the loss of momentum that would come from stretching the process out over a longer period 
of time. Ultimately, the value of including the mediators’ voices, and the further legitimacy those 
voices would give to the eventual outcome, overrode staff and board’s time concerns. 

So in the next few days, Tom, the director, contacted a number of the Dayton Mediation 
Center volunteer mediators and invited them to take part in the action evaluation. Within a couple of 
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weeks, nine volunteer mediators entered their “what” “why” and “how” responses into the 
database. They then gathered for their own feedback session, and arrived at the following 
consensus goals for their group: 

1. Provide a variety of types and levels of education and training opportunities, as well as 
educational resources, for volunteers and staff. 

2. Increase awareness of and referrals to Dayton Mediation Center through more publicity and 
outreach. 

3. Continue to be a fomm for resolving conflict in our community, especially among youth 
and neighbors. 

4. Evaluate impact of mediation Dayton Mediation Center services. 

5. Seek ways to broaden revenue producing opportunities,. 

6. Ensure personal safety for the staff, mediators, and clients in ways that are respectful to the 
clients and consistent with the Dayton Mediation Center mission. 

7. Increase involvement of the Dayton Mediation Center in providing services to the business 
community. 

8. Offer mediation training and resources to community groups, youth leadership groups, 
professional groups. 



From data analysis to action planning 

Once each stakeholding group has created a list of “consensus goals,” the action evaluator 
begins the next stage of data analysis, beginning by synthesizing the various groups’ goals into a 
single list of shared goals, and matching them with stakeholders’ “how” data. Then, bringing all 
stakeholders together in a single meeting, the action evaluator leads a process similar to that of the 
previous face-to-face meetings, in which all stakeholders agree to a single set of consensus goals. 

When the Dayton Mediation Center’s board/staff group came together with the volunteer 
mediators’ group, they came to consensus on five goals: 

1. Promote financial stability. 

2. Extend services to new markets. 

3. Increase referrals, visibility, and usage of Dayton Mediation Center. 

4. Enhance variety of types and levels of education and training opportunities for staff and 

volunteers. 

5. Undertake research and evaluation as part of our practice. 

After the group had articulated these goals, they divided into sub-groups to begin work on 
an action plan. 

It’s worth noting that at all stages of the action evaluation process, including this one, 
stakeholders have opportunities to learn not only about their own and each others’ goals and 
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motivations, but sometimes about the inaccuracy of their own preconceived notions. This increased 
awareness can enable stakeholders to think more creatively about how best to address the 
organization’s problems and work toward its goals. For example, one board member (a past 
volunteer mediator) has long brought his expertise as a businessman to the institution. His 
knowledge, especially of financial matters, has been of great value to the group. While dividing 
into subgroups to create an action plan, he noted that the volunteer mediators wouldn’t be 
interested in working on the first goal — promoting financial stability. “But,” one volunteer 
protested, “I’m interested.” Perhaps, he acknowledged, but most volunteers wouldn’t care. At that 
point, the action evaluator noted that when, earlier in the process, the participants had voted 
thumbs up, down, or sideways, on each goal, all eight volunteer mediators present had given full 
support to that goal, while only five of six board/staff members had. Faced with the evidence, the 
businessman could not help but see a capacity — and a resource — he might previously have 
overlooked. 

Once the sub-groups were ready to work, then, each took one goal, and prepared for it a 
list of “action items,” based primarily on the two groups’ combined “how” data. For example, the 
group responsible for the first goal — promote financial responsibility — produced a list of four 
action items: 

1 . Demonstrate to the city/management the value of mediation services. 

2. Formulate an ongoing grant writing strategy (local and national). 

3. Cultivate new revenue sources with other municipalities. 

4. Prepare a detailed revenue and line item budget. Use to control on a monthly basis. 

Each sub-group was also responsible for preparing a list of specific information to make 
the action items truly “actionable.” The first action item was fleshed out with the following: 

What: 1. Quantify the value of our services. 2. Newsletter targeted to funders. 3. 

Presentation to key players. 

Driving Forces: Quality (and enthusiasm) of presentations. 

Costs: Minimal cost in time. 

WhoAVhen: Tom lead person; George, Tim, and Bill committee members. George and 

staff — soon. 

Restraining Forces: Lack of staff and time. 

Benefits: 1. Continued commitment from city. 2. Basis to approach others for funding. 

After an hour or so, when all subgroups had prepared their action plans, they reconvened, 
posting their results on flip charts around the room for everyone to view. Then, at the suggestion 
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of the action evaluator, the group created a sixth goal — project management — “so we can 
maintain momentum and ensure we move from good ideas and energy to good implementation and 
action.” And together the group articulated a plan to make this goal actionable. 



Moving to Action 

Many a project has faltered at the very moment when a carefully crafted, eloquently 
articulated “action plan” is committed to paper. To avoid this all-too-common pitfall, the action 
evaluation process calls for specific individuals to be given specific responsibility for each step of 
the plan. And it endeavors to make sure that all are truly committed to the goals they have created 
together. 

Soon after the Dayton Mediation Center’s two groups worked out their action plan, a group 
made up of the Dayton Mediation Center’s four staff members, gathered to make plans for “follow- 
through.” One of their tasks was to get back in touch with action plan committee heads, to ensure 
that all were committed to doing the work they had agreed to do, and encourage them to move 
forward. 



Formative Stage 

Six weeks later, once implementation of the action plan has begun, committee heads will 
meet with the action evaluator to begin the “formative stage” of the action evaluation. During this 
time, the group will use the baseline data accumulated in the previous two months as a point of 
reference for self-conscious monitoring of progress, changes, and evolution. 

Over the course of the next several months — or years — this group will repeatedly assess 
whether each planned activity has been implemented and whether the activity has changed from the 
original plan. Because the action evaluation process required the stakeholders to create tangible 
“what” goals, this formative stage demands accountability from them. Did the committee 
responsible for the first goal — “promote financial responsibility” — do what they said they’d do? 
Did they quantify the value of The Center’s services? Did they launch a newsletter targeted to 
funders? Did they make presentations to key players? If not, why not? 

If they did follow through on their “what” goals, did the benefits they anticipated actually 
materialize? Can they determine specific ways to measure those benefits? 

At every step of the process, the action evaluation group will ask themselves what was 
learned, and how they have used that learning. 
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Summative Stage 



As a project reaches its conclusion, or a stock-taking point, participants use their evolved 
goals to establish criteria for retrospective assessment. Stakeholders will, for example, examine 
whether they have reached specified goals, and ask themselves, “why?” or “why not?” They will 
ask themselves how and what they could have done differently or better. 

It will be some time before the Dayton Mediation Center reaches this summative stage, but 
when they do, they will likely look back and see that some of their goals have evolved 
considerably. They may find that some of their goals remain the same, but also remain unmet. 
They’ll find that they’ve met some of the goals, but that the results were quite different from what 
they’d anticipated. Most importantly, they will have established internally relevant standards to 
assess their own success or to have their success externally assessed based on their internally 
derived agreements about their goals. 



Conclusion 

Action evaluation is more than an effective process of goal articulation and data gathering 
that systematizes what is normally done in the design and implementation of most conflict 
resolution and organizational development processes. The action evaluation process and content is 
enables participants to recognize the motivations, values, and interests necessary to negotiate 
consensus on shared goals and to promote reflexive evaluation among key stakeholders each step 
of the way as they move forward. 

Action evaluation is by no means a completed methodology. For example, action evaluation 
is designed to culminate in a summative process. However, due to many circumstances, most 
notably the time and energy poured into development of the methodology, we have only had three 
of some thirty projects reach that stage to date. Another gap between the aspirations and application 
of action evaluation so far is the limited input from foundations. We have been concerned with 
ensuring that those who fund conflict resolution in particular be directly integrated into goal setting 
by making explicit what they seek to accomplish and participating in dialogue about whether such 
goals are the most appropriate and if so how they may best be implemented, monitored and 
assessed (instead of more indirectly through funding choices). Yet to date only one foundation 
officer has provided data and participated in a feedback session (although three foundations have 
supported action evaluation projects). Finally, while action evaluation seeks to be useful and 
relevant across borders and has been applied on several continents, it is still primarily a U.S. -based 
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practice. We look forward to close collaboration with, and critique from, non-U.S. and non- 
Westem colleagues. 

While action evaluation is still new, the methodology is evolving-in-use and its application 
is undergoing further research and development, the need for such an integrated evaluation 
methodology is clearly an idea — and practice — whose time has come. We look forward to 
widening the circle of those who are using, and improving, action evaluation over the months and 
years ahead. 
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